Iterative, maximally probable, batch-mode commercial detection for audiovisual content

ABSTRACT

Identification of starting and ending times of commercial breaks and commercials within those commercial breaks found in audiovisual content is disclosed. A solution to a “batch optimization” problem is used in which commercial locations within a set of audiovisual content are detected as a group by choosing a set of commercial locations which optimizes a cost function which can include considerations of, for example, 1) one or more cues, 2) relative locations of commercials within the audiovisual content, and/or 3) probability models based on statistics obtained regarding characteristics of typical commercial and commercial breaks. Optimization can be done over the total set of commercial location decisions, rather than on a per-commercial basis. Additionally, the cost function can be iteratively evaluated and many more types of cues and combinations of cues can be used in detection of commercials.

CROSS REFERENCE TO OTHER APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.09/715,901, entitled ITERATIVE, MAXIMALLY PROBABLE, BATCH-MODECOMMERCIAL DETECTION FOR AUDIOVISUAL CONTENT filed Nov. 18, 2000 nowU.S. Pat. No. 6,993,245 which is incorporated herein by reference forall purposes, which claims priority to U.S. Provisional Application No.60/166,528, filed Nov. 18, 1999 which is incorporated herein byreference for all purposes.

BACKGROUND OF THE INVENTION

1. Field of the Invention

This invention relates to determining the starting and ending times ofcommercial breaks, as well as the starting and ending times ofcommercials within those commercial breaks, in audiovisual content(e.g., a television broadcast).

2. Related Art

There are a variety of previous approaches to detecting commercials in atelevision broadcast. However, previous approaches suffer from a flaw inthat they act as relatively simple finite state machines with little orno error recovery. Previous approaches sometimes make an erroneousdecision regarding a commercial location which only becomes apparent byconsidering data far ahead or after the commercial location in time.However, since previous approaches don't consider such data, informationregarding the erroneous decision is ignored and the error remainsuncorrected.

Merlino et al. of the MITRE corporation describe a multiple-cue-basedmethod for segmenting news programs, including finding the commercialbreaks. The Merlino et al. method uses black frames, audio silence andblank closed-captioning to find commercials, R. Lienhart et al. of theUniversity of Mannheim also describe a multiple-cue-based system fordetecting commercials. The Lienhart et al. method uses black frames,scene cuts and a measure of motion in a visual recording to detectcommercials. The Informedia project at Carnegie-Mellon University usedblack frames, scene cuts and lapses in closed-captioning to detectcommercials. Additionally, some VCRs come with commercial detectionbuilt in to the VCR. There are also a number of patents that describemethods for commercial detection, all of which use coincidences of blackframes, audio silence and/or certain closed-captioning signals to detectcommercials. For example, U.S. Pat. Nos. 4,319,286, 4,750,053,4,390,904, 4,782,401 and 4,602,297 detect commercials based on thesetypes of coincidences.

All of the previous approaches to commercial detection select commercialstart or end times as the times at which some combination of cues, suchas a black frame and an audio pause, coincide, with some optionalrestrictions. Typically, in previous approaches the decision as towhether or not a commercial starts at a particular time “t” isindependent of the analogous decision for any other time in theaudiovisual broadcast or recording. Previous approaches in which suchcomplete independence does not exist use only a very limited dependencein which the decision at time “t” may be affected by whether or not acommercial was thought to start within some window of time [t-n, . . . ,t] prior to the time “t”. Thus, in previous approaches, a commercialdetection decision made at any time “t” in a broadcast or recording isnot affected by parts of the broadcast or recording following time “t”and only sometimes is affected by limited parts (less than one minute)of the broadcast or recording immediately prior to time “t”. Further,none of the previous approaches have any sort of double-checking orerror recovery; once a decision is made for time “t”, by whateverheuristic the approach uses, the decision remains unchanged no matterwhat happens in the broadcast or recording after time “t” and no matterwhat other decisions are made before or after time “t”. In summary,previous approaches to commercial detection make decisions as tocommercial locations both time-locally and sequentially, i.e., only datafrom within a narrow time window about a particular time t is consideredin making the decision as to whether a commercial starts or ends at thattime t, and the decisions are made one at a time and are never reversed.

SUMMARY OF THE INVENTION

In accordance with the invention, the starting and ending times ofcommercial breaks, as well as the starting and ending times ofcommercials within those commercial breaks, can be found in audiovisualcontent (e.g., a television broadcast) using a method havingcharacteristics which overcome the above-described disadvantages of theprior art. The invention is implemented as a solution to a “batchoptimization” problem in which commercial locations within a set ofaudiovisual content are detected as a group by choosing a set ofcommercial locations which optimizes a cost function which can includeconsideration of, for example, 1) one or more of many types of visualrecording, audio recording and/or closed-captioning cues, 2) relativelocations of commercials within the audiovisual content, and/or 3)probability models based on statistics obtained regardingcharacteristics of typical commercial and commercial breaks (e.g.,commercial and commercial break duration, separation times ofcommercials and commercial breaks, likelihood of the presence of acommercial at any given time in a set of audiovisual content).Optimization can be done over the total set of commercial locationdecisions, rather than on a per-commercial basis. Additionally, the costfunction can be iteratively evaluated, increasing the accuracy ofcommercial location decisions produced by the method. The logic formaking decisions regarding detection of commercials in accordance withthe invention is orders of magnitude more sophisticated than that ofother approaches to commercial detection and produces correspondinglysuperior results. Additionally, many more types of information (cues)can be used in detection of commercials in accordance with the inventionthan have been used in other approaches to commercial detection.

The invention can make use of any one of a number of particular novelcollections of cues to enable commercial detection. Previous approachesto commercial detection have used some of the cues that can be used incommercial detection according to the invention, but have not used allof the cues or collections of cues that can be used in accordance withthe invention. Additionally, cues other than those specificallydescribed herein can be used to enable commercial detection according tothe invention.

Embodiments of the invention can have the following advantageouscharacteristics. The invention can be implemented so that commercialdetection is performed iteratively. The invention can be implemented sothat audiovisual content occurring before and/or after the time of apossible commercial is considered in deciding whether a commercial ispresent at that time. In particular, the invention can be implemented sothat commercials are detected in a set of audiovisual content byconsidering all of that set of audiovisual content. The invention can beimplemented so that the presence of multiple commercials in a set ofaudiovisual content can be detected at the same time. The invention canbe implemented so that a decision regarding whether a commercial isdetected at a particular location in a set of audiovisual content isdependent on the possible detection of one or more commercials at otherlocation(s) in the set of audiovisual content. In particular, theinvention can be implemented so that whether a commercial is detected ata particular location in a set of audiovisual content is dependent onall other possible detections of a commercial in the set of audiovisualcontent. Finally, as discussed above, the invention can be implementedso that commercial detection is based on the evaluation of a variety ofnovel cues or combinations of cues. Each of these aspects of theinvention produce improved commercial detection results as compared toprevious approaches to commercial detection.

The invention can be used for a wide variety of applications andpurposes, as will be appreciated by those skilled in the art in view ofthe description herein. For example, commercial detection in accordancewith the invention can be useful for enabling the observation ofaudiovisual content without commercial interruption, recordingaudiovisual content so that commercials are deleted from the recordedaudiovisual content, searching for a particular commercial or acommercial of a particular type within audiovisual content, andcustomizing commercials in audiovisual content. Additionally, theinvention can be used with, or implemented in, for example, conventionalnetwork television broadcasts, cable television broadcasts, televisionset-top boxes and digital VCRs.

BRIEF DESCRIPTION OF THE DRAWINGS

Various embodiments of the invention are disclosed in the followingdetailed description and the accompanying drawings.

FIG. 1 is a flow chart of a method according to the invention.

FIG. 2 is a flow chart of a method according to a particular embodimentof the invention.

FIG. 3A is a graph of an example of a function P(t) which indicates thelikelihood that a commercial starts or ends at each time during a set ofaudiovisual content.

FIG. 3B is a graph of an example of a function S(t) which represents aprobability model of the likely location, relative to a particularcommercial start or end time, of other commercial start or end times.

FIG. 3C is a graph of a function C(t) produced by convolving thefunction P(t) of FIG. 3A with the function S(t) of FIG. 3B.

FIG. 3D is a graph of an example of a function R(t) which represents aprobability model of the likelihood, at all times within a set ofaudiovisual content, that a commercial is in progress.

FIG. 3E is a graph of a function P′(t) produced by point-wisemultiplying the function C(t) of FIG. 3C by the function R(t) of FIG.3D.

FIG. 3F is a graph of an example of a function L(t) which represents aprobability model of the typical duration of a commercial break.

FIG. 3G is a graph of an example of a function W(t) which represents aprobability model of the typical time separation between commercialbreaks in a set of audiovisual content.

DETAILED DESCRIPTION

The invention can be implemented in numerous ways, including as aprocess, an apparatus, a system, a composition of matter, a computerreadable medium such as a computer readable storage medium or a computernetwork wherein program instructions are sent over optical or electroniccommunication links. In this specification, these implementations, orany other form that the invention may take, may be referred to astechniques. A component such as a processor or a memory described asbeing configured to perform a task includes both a general componentthat is temporarily configured to perform the task at a given time or aspecific component that is manufactured to perform the task. In general,the order of the steps of disclosed processes may be altered within thescope of the invention.

A detailed description of one or more embodiments of the invention isprovided below along with accompanying figures that illustrate theprinciples of the invention. The invention is described in connectionwith such embodiments, but the invention is not limited to anyembodiment. The scope of the invention is limited only by the claims andthe invention encompasses numerous alternatives, modifications andequivalents. Numerous specific details are set forth in the followingdescription in order to provide a thorough understanding of theinvention. These details are provided for the purpose of example and theinvention may be practiced according to the claims without some or allof these specific details. For the purpose of clarity, technicalmaterial that is known in the technical fields related to the inventionhas not been described in detail so that the invention is notunnecessarily obscured.

FIG. 1 is a flow chart of a method 100 according to the invention fordetecting one or more commercial breaks in a set of audiovisual content,each commercial break including one or more commercials. The method 100identifies starting and ending times for each commercial break, as wellas starting times for each commercial within a commercial break.

In step 101 of the method 100, the data representing the audiovisualcontent (including closed-captioning or other transcription data, ifapplicable) is analyzed to identify the presence of one or morepredetermined types of cues. The location and duration of each cue(i.e., the beginning and ending times of each cue) can be determined.Other characteristics of the cues can be determined as well. Examples ofthe types of cues that can be identified are discussed in more detailbelow.

In step 102 of the method 100, the cues are analyzed to identifypossible locations of commercial beginnings or endings (candidate times)within the audiovisual content. Examples of such analysis are discussedfurther below. A score is assigned to each candidate time, the scorerepresenting a probability that the candidate time is in fact abeginning or ending of a commercial.

In step 103 of the method 100, the scores associated with each candidatetime are adjusted. The score for a candidate time can be adjusted, forexample, based on an analysis of one or more cues proximate to thecandidate time that are different than the one or more cues used toidentify the candidate time. For instance, as described in more detailbelow with respect to step 206 of the method 200 (see FIG. 2) the scoreassociated with a candidate time can be adjusted based on the presenceor absence of one or more cues within a specified time window thatincludes the candidate time or to which the candidate time issufficiently proximate (i.e., is less than a specified short amount oftime, such as several seconds, before or after the time window). Thescore for a candidate time can also be adjusted, for example, based onan evaluation of the relationship between the candidate time and one ormore other candidate times. In particular, as described in more detailbelow with respect to step 207 of the method 200 (see FIG. 2), thislatter type of adjustment can make use of one or more probability modelsthat describe expected relationship(s) between a candidate time and theone or more other candidate times.

The invention can be implemented so that after the adjustment of scoresin step 103, scores below a specified threshold are eliminated. However,this need not necessarily be done.

In step 104 of the method 100, one or more commercial 30 breaks areconstructed based on an evaluation of the adjusted scores of thecandidate times and relationships among the candidate times. Inparticular, as described in more detail below with respect to step 208of the method 200 (see FIG. 2), the evaluation of relationships amongcandidate times can make use of one or more probability models thatdescribe expected relationship(s) between candidate times.

The method 100 can be implemented so that step 104 is iterativelyperformed. This can improve the identification of commercial breaks andindividual commercials in a set of audiovisual content by enablingreconsideration of high-scoring candidate times which were not chosen asa commercial start or end time in an earlier performance of the step104.

FIG. 2 is a flow chart of a method 200, according to a particularembodiment of the invention, which can be implemented in a system,apparatus or computer program according to the invention to accomplishcommercial detection in accordance with the invention. The data input tothe method 200 represents audiovisual content (e.g., a televisionbroadcast). Herein, “audiovisual content” includes one or both of visualdata and audio data, and can also include closed-captioning data. Insome embodiments of the invention, the input data is stored on a datastorage medium or media, such as, for example, a computer hard disk(DRAM or SRAM), a CD-ROM, a DVD disk or a VHS tape. As described below,in other embodiments of the invention, the input data can represent live(i.e., not stored) audiovisual content (e.g., a live televisionbroadcast) that is acquired in real time. As shown in FIG. 2, the inputdata is compressed, however this need not necessarily be the case.

The output of the method 200 is a list of commercial break start and endtimes within the audiovisual content, plus lists of the individualcommercial start times (and, therefore, implicitly, the individualcommercial end times) within each commercial break. The identificationof the start and end times of commercial breaks and individualcommercials in the audiovisual content can be used to, for example,enable editing of the audiovisual content. For instance, the detectedcommercials can be deleted from the audiovisual content or thecommercials can be altered in a desired manner. (The modified data,e.g., data representing commercial-less audiovisual content, can bestored on a data storage medium or media). However, a method accordingto the invention need not be used to edit the audiovisual content withinwhich the commercial breaks and individual commercials have beendetected.

In the first step of the method 200, step 201, the input data isidentified as the input data is input to the method 200 (e.g., read froma data storage medium or media). Apparatus for effecting suchidentification is known to those skilled in the art and will depend onthe source of the input data. For example, readers for all useful datastorage media are readily available.

The next step of the method 200, step 202, is to decompress the rawdata, if necessary. The invention does not require the original inputdata to be in compressed format, nor does the invention require theinput data to be in uncompressed format. However, if the input data isin compressed format, the implementation of the invention illustrated inFIG. 2 requires a decompression mechanism. The precise form of thedecompression mechanism depends on the compression format, butdecompression mechanisms for all useful forms of compression formats arereadily available.

In step 203 of the method 200, the decompressed data is split intovisual, audio and closed-captioning subcomponents. (As indicated above,one or two of visual, audio and closed captioning data may not bepresent as part of the input data.) The precise form of the mechanismfor splitting apart the input data can depend on the decompressionmechanism used, but such data-splitting mechanisms are readily availablefor any useful compression format.

In step 204, the audiovisual content is evaluated to identify thepresence of one or more cues in the audiovisual content. Each datasubcomponent produced by step 203 is input into one or more analyzers.The analyzer(s) identify the location(s) and duration(s) of cue(s)within the audiovisual content. In particular, the presence of cue(s)throughout an entire set of audiovisual content can be identified. Theanalzyers may also identify other characteristics of the cue(s). Thefollowing is an exemplary list of characteristics of a set ofaudiovisual content regarding which cues can be identified within theset of audiovisual content: 1) an audio pause (i.e., a period of silenceor near silence) in the audio content, 2) a sequence of black frames inthe visual content, 3) a scene cut or fade in the visual content, 4) asignificant (i.e., greater than a specified amount) change in averagevolume in the audio content, 5) the presence of music in the audiocontent, 6) speaker identity, 7) the “density” of scene breaks (cuts) orfades in the visual content (i.e., the number of scene breaks and/orscene fades during a specified time window divided by the duration ofthe time window), 8) the absence of a usually present network icon(whose shape and color characteristics can be learned automatically byappropriately analyzing a region, e.g., a region, such as a corner, nearthe edge of the visual content, of the visual content in which a networkicon is expected to be present, 9) the degree of motion in a period ofvisual content, 10) the presence of text in the visual content, 11) theoccurrence of specified closed-captioning formatting signals, and 12)the absence of closed-captioning. Suitable methods for identifying eachof the above-listed cues have been published in various academicjournals, industry journals and conference proceedings, and are known tothose skilled in the art. As will be appreciated by those skilled in theart in view of the description herein, other cues not listed above canalso be used, alone or in combination with each other and/or one or moreof the above listed cues to enable commercial detection according to theinvention. The specific methods used to identify the above listed orother cues may affect the overall performance of a system or methodaccording to the invention, but, in general, any such methods can beused with the invention. However, it is an advantageous aspect of theinvention that the invention enables use of a combination of theabove-listed cues to effect commercial detection.

In an alternative implementation of the invention, the output from step201 of the method 200 would be input directly to some or all of theanalysis engines of step 204. That is, some of the analysis engines canbe made to operate directly on the compressed data, depending on thecompression format. For example, the black-frame detection, scene-cutdetection, motion analysis and audio-level change detection can alloperate directly on data that has been compressed in the MPEG-1 orMPEG-2 format.

In step 205 of the method 200, one or more of the cues identified instep 204 are analyzed to identify candidate times within the audiovisualcontent at which a commercial beginning or a commercial ending mayoccur. For example, an audio pause often accompanies either thebeginning or the end of a commercial, so the presence of an audio pausein the audio content can be identified as a factor that militates towardestablishing a candidate time at some time during or proximate to theaudio pause. Similarly, a sequence of black frames often accompanieseither the beginning or the end of a commercial, so the presence of asequence of black frames in the visual content can be identified as afactor that militates toward establishing a candidate time at some timeduring or proximate to the sequence of black frames. A scene cut or fadealso typically accompanies the beginning or the end of a commercial, sothe presence of a scene break or fade in the visual content can beidentified as a factor that militates toward establishing a candidatetime at some time during or proximate to the scene break or fade. Thebeginning and end of a commercial break are often accompanied by anoticeable increase and decrease in volume, respectively, so that asignificant change in average volume (measured over a specified windowof time) can be identified as a factor that militates towardestablishing a candidate time at some time proximate to times at whichthe volume is seen to change significantly. Commercials often includerelatively more musical content than the rest of a set of audiovisualcontent, so the occurrence of a time window of specified duration (e.g.,the expected duration of a typical commercial break, such as 60 seconds,or the expected duration of a typical commercial, such as 15 or 30seconds) having relatively high musical content (e.g., relatively highdensity of musical content relative to the density of musical content inother, proximate time windows) can be identified as a factor thatmilitates toward establishing candidate times at the beginning and endof such a time window. The beginning or end of a commercial is oftenaccompanied by a change in speaker identity, so the occurrence of achange in speaker identity can be identified as a factor that militatestoward establishing a candidate time at, or proximate to, the time atwhich such a change in speaker identity occurs. A commercial break oftenincludes a relatively high density of scene breaks and/or fades (since ascene break or fade typically occurs at the beginning and end of acommercial break, as well as at the transition between commercialswithin a commercial break, and since commercials often include arelatively large number of scene breaks and/or fades per unit timewithin the commercial), so the occurrence of a time window of aspecified duration (e.g., 60 seconds) during which the density of scenebreaks and/or scene fades is relatively high (i.e., exceeds a specifiedthreshold), or a significant change in density of scene breaks and/orscene fades over one window of time with respect to a proximate windowof time, can be identified as a factor that militates towardestablishing candidate times at the beginning and end of such a timewindow. A network icon is sometimes present during the noncommercialparts of a television broadcast; therefore, if a network icon isdetermined to be present in a set of audiovisual content, thedisappearance of the network icon typically accompanies the beginning ofa commercial break and the appearance of the network icon typicallyaccompanies the end of a commercial break, so the appearance ordisappearance of a network icon can be identified as a factor thatmilitates toward establishing a candidate time at, or proximate to, atime at which the network icon appears or disappears. Since the averagemotion level in the visual content of a commercial is oftensignificantly different than the average motion level of other visualcontent in a set of audiovisual content, significant change in theamount of motion in the visual content of a time window (e.g., about 60seconds) relative to the amount of motion in the visual content in aproximate time window can be identified as a factor that militatestoward establishing candidate times at, or proximate to, the beginningand end of such a time window. The appearance of text (other thanclosed-captioning) in a set of audiovisual content often accompanies thebeginning of a commercial break and the disappearance of text oftenaccompanies the end of a commercial break, so the appearance ordisappearance in a set of audiovisual content of text other thanclosed-captioning can be identified as a factor that militates towardestablishing a candidate time at, or proximate to, a time at which textappears or disappears. If closed-captioning data is present in the datarepresenting the audiovisual content, a closed-captioning scrollingformat change often occurs at the beginning or the end of a commercialbreak, so the occurrence of a closed-captioning scrolling format changecan be identified as a factor that militates toward establishing acandidate time at, or proximate to, the time at which such a formatchange occurs. If closed-captioning data is present in the datarepresenting the audiovisual content, the disappearance ofclosed-captioning often accompanies the beginning of a commercial breakand the appearance of closed-captioning often accompanies the end of acommercial break, so the appearance or disappearance ofclosed-captioning can be identified as a factor that militates towardestablishing a candidate time at, or proximate to, a time at whichclosed-captioning appears or disappears.

As indicated above, it is an advantageous aspect of the invention thatthe invention enables use of a combination of the cues to effectcommercial detection. In particular, the invention can enable the use ofcues and combinations of cues that have not previously been used forcommercial detection. For example, the invention can advantageouslyenable any one of detection of the absence of a network icon, ananalysis of musical content present in a set of audiovisual content, thedensity of scene breaks and/or fades, or analysis of the identity ofspeakers of spoken content to be used alone as a commercial detectioncue. These cues can also be used in any combination with each other orany other cue. In particular, it is anticipated that one or more ofthese cues can advantageously be used in combination with one or more ofthe following cues: 1) the occurrence of an audio pause, 2) theoccurrence of a sequence of black frames, 3) a scene cut or fade, 4) theoccurrence of specified closed-captioning formatting signals, and 5) theappearance or disappearance of closed-captioning.

Step 205 outputs a list of candidate times at which commercials may bebeginning or ending, together with a score or probability associatedwith each candidate time. In one implementation of the invention, eachcandidate time is assigned the same initial score. Alternatively, thescores assigned to candidate times can vary. For example, the score fora candidate time can depend on which cue(s) were used to identify thecandidate time. The beginning or end of a commercial can be deduced fromthe presence of some cues with a greater degree of confidence than thatassociated with the presence of other cues. To the extent that acandidate time is identified based on a cue with which a relatively highdegree of predictive confidence is associated, the score assigned tothat candidate time can be relatively higher than would be the case if arelatively low degree of predictive confidence was associated with thecue. Additionally, the score for each candidate time can be dependent onhow strongly the cue is present in the audiovisual content, asdetermined in accordance with a criterion or criteria appropriate forthat cue: the more strongly a cue is present, the higher the score. Forexample, when one of the cues used to establish a candidate time is anaudio pause, the score established for the candidate time can bedependent on the duration of the audio pause and/or the degree ofsilence during the audio pause (e.g., the score for the candidate timeis made relatively greater the longer the audio pause or the less soundthat is present during the audio pause). Or, for example, when one ofthe cues used to establish a candidate time is a sequence of blackframes, the score established for the candidate time can be dependent onthe duration of the sequence of black frames and/or the completeness ofthe blackness of the frames (e.g., the score for the candidate time ismade relatively greater the longer or blacker the sequence of blackframes). Or, for example, when one of the cues used to establish acandidate time is a scene cut, the score established for the candidatetime can be dependent on the number of pixels that changed by more thana threshold amount from one frame to another (e.g., the score for thecandidate time is made relatively greater as more pixels changed betweenscenes) and/or dependent on the total change of all the pixels from oneframe to another (where the “change” for each pixel is the change in thecolor or other components of a pixel). Or, for example, when one of thecues used to establish a candidate time is a significant average audiovolume change, the score established for the candidate time can bedependent on degree of the volume change (e.g., the score for thecandidate time is made relatively greater as degree of the volume changeincreases). Those skilled in the art can readily appreciate how thescore for a candidate time can be adjusted based on aspects of othercues present in the audiovisual content proximate to the candidate time.Additionally, the score for a candidate time can be dependent on theconfidence level associated with identification of the cue in theaudiovisual content: the greater the confidence level, the higher thescore. (This confidence level is different than the confidence levelassociated with the predictive capability of the cue, discussed above.)For example, sound represented in audio data may be sound in the audiocontent or noise. The score for a candidate time identified at least inpart based on the presence of an audio pause can be increased ordecreased in accordance with extent to which the degree of noise presentin the audio data increases or decreases the confidence with which anaudio pause can be detected.

In step 206 of the method 200, the scores associated with each candidatetime can be adjusted based on the presence or absence of one or morecues within some time window proximate to the candidate time. The cue(s)used to adjust the score of a candidate time in step 206 are differentthan the cue(s) used to establish the candidate time and an initialassociated score in step 205. The duration of the time window andlocation of the time window with respect to the cue is dependent on thetype of cue. For instance, the score for a candidate time is increased(i.e., the likelihood that the candidate time correctly indicates thebeginning or ending of a commercial is deemed to increase) in each ofthe following cases: 1) the candidate time is coincident with the timeat which an audio pause (which is a window of audio silence or nearsilence) occurs, 2) the candidate time is within or sufficientlyproximate to a time window in which the closed-captioning scrollingformat is different from that which is typical for audiovisual contentof this type, 3) the candidate time is within or sufficiently proximateto a time window during which closed-captioning is absent (foraudiovisual content that is known to be closed-captioned), 4) thecandidate time is within or sufficiently proximate to a time window ofat least a specified duration (e.g., 60 seconds) and including highmusical content, 5) the candidate time is within or sufficientlyproximate to a time window during which the density of scene breaksand/or scene fades exceeds a specified threshold, 6) the candidate timeis sufficiently proximate to a time window of at least a specifiedduration (e.g., 0.5 seconds) and in which the average motion in thevisual content, measured in a specified manner, is less than a specifiedthreshold, 7) the candidate time is within a time window during which anetwork icon (which has been found to be persistent through a majorityof the visual content) is not present at a specified location within thevisual content (e.g., a region, such as a corner, near the edge of thevisual content), 8) the candidate time is very near (e.g., within about2 seconds) a time at which the time-averaged audio volume (averaged overa time window of about 10 seconds) has changed by a magnitude of greaterthan a specified threshold, 9) the candidate time is sufficientlyproximate to (within about 1 second) a time when text is present in thevisual content, 10) the candidate time is within a specified duration oftime (e-g., a few seconds) after the presence in the closed-captioningstream of certain keywords or phrases such as “commercial”, “break”,“coming up” or “after”, or within a specified duration of time (e.g., afew seconds) prior to the presence in the closed-captioning stream ofcertain keywords or phrases such as “welcome”, “hello” or “we're back,11) the candidate time is within a specified duration of time (e.g., 2seconds) from a time at which the speaker identity has changed, and 12)the candidate time is within a specified duration of time (e.g., one toseveral seconds) from a time window of greater than a specified duration(e-g., 1 minute) that does not include speech from a speaker whosespeech has been determined to be present in the audiovisual content withgreater than a specified frequency. The amount by which a score isadjusted can be dependent on the same types of analyses done toestablish an initial score for a candidate time, as described above withrespect to step 205. (However, the particular analyses done in step 206need not, but can be, the same as those done in step 205.) Inparticular, the amount of the adjustment to a score for a candidate timecan be dependent on how strongly the cue is present in the audiovisualcontent, as determined in accordance with a criterion or criteriaappropriate for that cue: in general, the more strongly a cue ispresent, the greater the adjustment to the score. Additionally, theamount of the adjustment to a score for a candidate time can bedependent on how high or low the score is prior to the adjustment. Forexample, a cue that strongly indicates the presence of a commercialbeginning or ending may cause a larger adjustment in a relatively lowscore than in a relatively high score. The particular quantities,keywords, and other algorithm parameters given above are illustrative;they may be changed, within appropriate constraints, as can beappreciated by those skilled in the art, without adversely affecting theoperation of the invention.

In step 207 of the method 200, the scores associated with each candidatetime are further adjusted based on one or more probability models ofcharacteristic(s) of the occurrence of commercials and/or commercialbreaks within audiovisual content. For example, the scores of thecandidate times can be adjusted based on a probability model of thetime-separation of commercial start and end times. The scores of thecandidate times can also be adjusted based on a probability model of thetypical locations of commercial breaks within audiovisual content. Sucha probability model can be constructed by collecting statisticsregarding the relevant characteristic(s) across many sets of audiovisualcontent of a variety of different types, in order to produce a genericprobability model that applies to all types of audiovisual content. Sucha probability model can also be made specific to a particular type ortypes of audiovisual content (including a particular audiovisualprogram) by only combining statistics across audiovisual content ofthose type(s). This can be desirable to increase the accuracy obtainedwhen the probability model is used to aid in detection of commercials inaudiovisual content of those type(s). Finally, such a probability modelcan be constructed manually based on the intuition of the implementer ofthe model as to the characteristic being modeled (e.g., how longcommercials typically last or when commercial breaks tend to occur ingiven audiovisual content). However constructed, the probabilitymodel(s) can be represented as functions of time, as described below.

In step 207, one or more probability models can be applied to the listof score-adjusted candidate times generated by step 206 to furtheradjust the scores of the candidate times. The list of score-adjustedcandidate times generated by step 206 is first represented as a functionof time, P(t), which indicates the likelihood that a commercial startsor ends at each time during the audiovisual content. For all candidatetimes, P(t) can be made equal to the adjusted score associated with thatcandidate time (perhaps normalized by the total of all the adjustedscores), while for all other times P(t) can be made equal to zero. FIG.3A is a graph of an example of a function P(t).

A function S(t) is determined, representing a probability model of thelikely location, relative to a particular commercial start or end time,of other commercial start or end times. The function S(t) can beparticularized to be representative of a particular type or types ofaudiovisual content. For example, for American television programs, thefunction S(t) will have peaks around +/−30 seconds, +/−60 seconds and+/−15 seconds. FIG. 3B is a graph of an example of a function S(t).

P(t) is convolved with S(t) to produce a function C(t). The value ofC(t) will be very high at times at which a high value of P(t) isseparated from other times having a high value of P(t) by time durationsS(t) having a high value. In other words, C(t) will be very large atvalues of t which are high-scoring candidate times and where there areother high-scoring candidate times before and/or after t by an amount oftime corresponding to a typical commercial duration (e.g., 30 seconds).FIG. 3C is a graph of a function C(t) produced by convolving thefunction P(t) of FIG. 3A with the function S(t) of FIG. 3B.

A function R(t) is determined, representing a probability model of thelikelihood, at all times within a set of audiovisual content, that acommercial is in progress. If the probability model is restricted to bebased on a particular class of well-defined sets of audiovisual content(e.g., the different versions of a particular recurring audiovisualprogram) and if commercials tend to be placed at approximately the sametimes in each such set of audiovisual content, then the probabilitymodel will have well-defined zones during which the probability of acommercial being in progress is high. On the other hand, if theaudiovisual content on which the probability model is based is notrestricted at all, so that the probability model is learned across alltypes of audiovisual content, the probability model will likely be closeto a uniform distribution (a flat function), which is not very useful.Therefore, it is desirable to restrict the audiovisual content on whichthe probability model is based. In particular, it is desirable to basethe probability model which the function R(t) represents on audiovisualcontent which is similar to that in which commercials are to bedetected. FIG. 3D is a graph of an example of a function R(t).

As indicated above, step 207 begins by convolving the function P(t) withthe function S(t) to produce the function C(t). The function C(t) isthen point-wise multiplied by the function R(t) to produce a functionP′(t). FIG. 3E is a graph of a function P′(t) produced by point-wisemultiplying the function C(t) of FIG. 3C by the function R(t) of FIG.3D. The function P′(t) is resampled at the candidate times: thesesamples represent further adjusted scores for the candidate times.

The method 200 is described above as including both score adjustments ofthe type described in step 206 and score adjustments of the typedescribed in step 207. However, the method 200 could be implemented withonly one of those types of score adjustments, i.e., the method 200 couldinclude only one of steps 206 and 207.

After the adjustment of scores in step 207, optionally, scores below aspecified threshold can be eliminated.

In step 208 of the method 200, the candidate times and associatedadjusted scores are evaluated, and starting and ending times forcommercial breaks and individual commercials within those commercialbreaks are identified based on that evaluation. Two additionalprobability models (which can be represented as functions of time) areused in this evaluation: 1) a function L(t), which represents aprobability model of the typical duration of a commercial break, and 2)a function W(t), which represents a probability model of the typicaltime separation between commercial breaks. Each of these probabilitymodels, as the probability models discussed above with respect to step207, can be constructed based on statistics collected across many typesof audiovisual content or across only a particular type or types ofaudiovisual content, or can be constructed based on the intuition of theimplementer of the model regarding the characteristic being modeled.FIG. 3F is a graph of an example of a function L(t) and FIG. 3G is agraph of an example of a function W(t).

Step 208 begins by selecting the candidate time with the highest scoreto be a commercial start or end time (whether that time is a start timeor end time is unknown at this point). A commercial break is thenconstructed based on the selected candidate time by successivelyevaluating candidate times in order of decreasing score and addingcandidate times to the commercial break that satisfy each of thefollowing criteria: 1) the additional candidate time is well-spaced intime, in accordance with the function S(t), from each candidate timethat has already been included in the commercial break, 2) theadditional candidate time does not create a commercial break which istoo long, in accordance with the function L(t), and 3) the additionalcandidate time is not too close to other existing commercial breaks, inaccordance with the function W(t), that have already been identified bythe step 208. Stated another way, candidate times continue to be addedto a commercial break in order of score as long as there are anycandidate times for which all of the following are true: 1) the value ofS(t), where “t” is the time separation between the candidate time beingevaluated and a candidate time already included in the commercial break,is above a specified threshold value for each candidate time alreadyincluded in the commercial break, 2) the value of L(t), where “t” is theduration of the commercial break if the candidate time is added to thecommercial break, is not below a specified threshold value, and 3) thevalue of W(t), where “t” is the time separation between the candidatetime and an existing commercial break, is not below a specifiedthreshold value for each existing commercial break.

Once no more candidate times can be added to the commercial breakcurrently being constructed, step 208 attempts to find a new candidatetime around which to construct a new commercial break. The candidatetime with the highest score above a specified threshold that is notcurrently part of any commercial break, and that is separated from allexisting commercial breaks by a time “t” for which W(t) is above aspecified threshold, is selected as a new candidate time upon which tobase the construction of a new commercial break. If such a candidatetime is identified, then the construction of a commercial break proceedsas described above. If no such candidate time can be identified, thenstep 208 terminates. The output of step 208 is a list of commercialbreak start and end times, and lists of start times of individualcommercials within each commercial break.

Finally, step 209 of the method 200 causes the set of decisions made instep 208 regarding start and end times of commercial breaks andcommercials to be iteratively refined and optimized. Step 209 attemptsto account for other high-scoring candidate times which may occur in ornear a previously identified commercial break, but which have not yetbeen chosen as a commercial start or end time. In step 209, thefollowing procedure is performed for each candidate time which has notyet been selected as a commercial start or end time and for which Pt(t)has a magnitude above a specified threshold value.

First, the candidate time is added to the most temporally proximatecommercial break, provided that the candidate time would not cause thatcommercial break to become too long (i.e., the value of L(t), where “t”is the duration of the commercial break if the candidate time is addedto the commercial break, is not below a specified threshold value) andwould not cause that commercial break to be too close to anothercommercial break (i.e., the value of W(t), where “t” is the timeseparation between the candidate time and an existing commercial break,is not below a specified threshold value for each existing commercialbreak).

Next, additional candidate times that are not part of the commercialbreak to which a candidate time has just been added are evaluatedaccording to the same criteria, described above, that were used toconstruct the commercial break in the first place. That is, theprobability models S(t), L(t), and W(t) are all considered as describedabove.

The new commercial break may have too many candidate times within it.Thus, an attempt is made to remove candidate times that are located tooclosely together. For each candidate time of the commercial break, acomputation is made of the average S(t) for the time separation betweenthe candidate time and the two adjacent candidate times. If the averageis below a specified threshold, then it may be that either the candidatetime or the adjacent candidate times are not accurate commercial startor end times. If the P′(t) score of the candidate time is lower than theaverage P′(t) of the two adjacent candidate times, the candidate time iseliminated from the commercial break. Otherwise, any of the two adjacentcandidate times that are within a specified small time separation (e.g.,25 seconds) of the candidate time are eliminated from the commercialbreak.

Finally, after the addition(s) and elimination(s) of candidate timeshave been made, a new average score is calculated for the candidatetimes of the commercial break. If the average score is higher than theaverage score for the candidate times of the commercial break beforechanges were made in step 209, the changes are kept. Otherwise, thechanges are discarded and the candidate times of the commercial breakrevert back to the candidate times before the addition(s) andelimination(s) made in step 209.

After an attempt has been made to include each of the candidate timesfor which P′(t) is above a threshold, a check is made to see if any ofthese candidate times were in fact included in a commercial break. Ifnone were, then step 209 terminates and outputs the finaldetermination(s) of the method 200 regarding the start and end times ofcommercial breaks and individual commercials in the audiovisual content.If any of these candidate times were included in a commercial break,then step 209 is repeated: for each candidate time having a score P′(t)above a specified threshold, an attempt is made to add that candidatetime to the nearest commercial break. Many iterations of step 209 may berequired before a stable solution is produced (i.e., further changes arenot made to the determined start and end times of commercial breaks andindividual commercials in the audiovisual content).

The invention as described above can be modified slightly in order tooperate on audiovisual content that is being received live via a tuner,cable or other means, and to produce commercial detection results withminimal delay relative to the current live position within theaudiovisual content. If the cue(s) are not transmitted with the datarepresenting the audiovisual content, to accomplish this, the inventionis implemented to enable the analysis of the audiovisual content toidentify the presence of cues (the invention can be implemented toidentify one, some or all of the cues described above with respect tostep 204 of the method 200 of FIG. 2) at a rate that is at least as fastas the rate at which the data representing the audiovisual content isbeing received. The invention is further implemented so that this can bedone while receiving the transmission of the data representing theaudiovisual content. The invention is implemented to evaluate lists ofidentified cues over a window from the present time “t” back to a time“T” seconds into the past. If the computations done in steps 205 through209 of the method 200 of FIG. 2 (or comparable steps of another methodaccording to the invention, such as the steps 102 through 104 of themethod 100 of FIG. 1) can be done in N seconds, a new set of commerciallocation estimates up to the current time “t” minus approximately Nseconds can be produced. This can be done every N seconds. For instance,if N=0.5 seconds, the commercial locations up to the current time with0.5 seconds delay can be computed and this can be done every 0.5seconds. This would be sufficiently fast for commercial detectionapplications that require actions to be taken in roughly “real-time,”e.g., changing the channel or stopping recording when a commercialbegins. The accuracy of such an embodiment of the invention may not beas high as for an embodiment that operates on audiovisual content thatis stored in its entirety, since, in the latter case, information beyondtime “t” can be used to improve the commercial detection decision attime “t”.

The invention can be implemented entirely in software, entirely inhardware (e.g., using DSPs and/or special purpose ASICs) or in acombination of the two. Firmware can also be used to implement some partor all of the invention.

The invention can be used for a wide variety of applications, as can beappreciated by those skilled in the art in view of the descriptionherein. In general, the invention can be used with any broadcast orother data transmission over a network (e.g., conventional networktelevision broadcasts, cable television broadcasts, broadcasts ortransmissions over a computer network such as the Internet—and, inparticular, the World Wide Web portion of the Internet). Additionally,the invention can be used generally to detect commercials in audiovisualcontent represented by any type of data, which data can be stored on adata storage medium or media, or provided to a system or methodaccording to the invention in real time. Further, the invention can beimplemented in a wide variety of apparatus, as can also be appreciatedby those skilled in the art in view of the description herein, such as,for example, television set-top boxes, digital VCRs, computers(including desktop, portable or handheld computers) or any of a varietyof other computational devices (including many which are now being, orwill in the future be, developed).

Various embodiments of the invention have been described. Thedescriptions are intended to be illustrative, not limitative. Thus, itwill be apparent to one skilled in the art that certain modificationsmay be made to the invention as described herein without departing fromthe scope of the claims set out below.

Although the foregoing embodiments have been described in some detailfor purposes of clarity of understanding, the invention is not limitedto the details provided. There are many alternative ways of implementingthe invention. The disclosed embodiments are illustrative and notrestrictive.

1. A method of processing an audiovisual signal containing audiovisualcontent, comprising: identifying a set of candidate times in theaudiovisual content contained in the audiovisual signal, wherein each ofthe set of candidate times corresponds to a potential commercialboundary and the set of candidate times includes at least a firstcandidate time; determining a probability based at least in part oninformation associated with the first candidate time in the context ofat least one member of a set of identified commercial boundaries;determining whether the probability satisfies a threshold; and includingthe first candidate time in the set of identified commercial boundarieswhen the probability is determined to satisfy the threshold.
 2. A methodas recited in claim 1, wherein the audiovisual signal is obtained from astorage device.
 3. A method as recited in claim 1, wherein theaudiovisual signal is transmitted.
 4. A method as recited in claim 1,wherein the first candidate time occurs earlier in the audiovisualcontent compared to the at least one member of the set of identifiedcommercial boundaries.
 5. A method as recited in claim 1 furtherincluding generating a new audiovisual signal, wherein the newaudiovisual signal does not include a commercial identified by the setof identified commercial boundaries.
 6. A method as recited in claim 1further including displaying the audiovisual signal with a commercialidentified by the set of identified commercial boundaries removed.
 7. Amethod as recited in claim 1 further including displaying a commercialidentified by the set of identified commercial boundaries.
 8. A methodas recited in claim 1, wherein content of a commercial included in theaudiovisual content is unknown during processing of the audiovisualsignal.
 9. A method as recited in claim 1, wherein an identifiedcommercial boundary may be removed from the set of identified commercialboundaries.
 10. A method as recited in claim 1 further includingdetermining a score for each of the set of candidate times.
 11. A methodas recited in claim 1 further including determining a score for each ofthe set of candidate times, including using a cue.
 12. A method asrecited in claim 1 further including determining a score for each of theset of candidate times, including using a cue associated with videocontent.
 13. A method as recited in claim 1 further includingdetermining a score for each of the set of candidate times, includingusing a cue associated with audio content.
 14. A method as recited inclaim 1 further including determining a score for each of the set ofcandidate times, including using a cue associated with close captioningcontent.
 15. A method as recited in claim 1, wherein the probability isfurther based at least in part on statistical information associatedwith historical commercial data.
 16. A method as recited in claim 1,wherein the probability is associated with a duration of a commercial.17. A method as recited in claim 1, wherein the probability isassociated with a duration of a commercial break.
 18. A method asrecited in claim 1, wherein the probability is associated with a timebetween a first commercial break and a second commercial break.
 19. Amethod as recited in claim 1, wherein the probability is associated witha location of a commercial break in a program.
 20. A system forprocessing an audiovisual signal containing audiovisual content,comprising: a processor configured to: identify a set of candidate timesin the audiovisual content contained in the audiovisual signal, whereineach of the set of candidate times corresponds to a potential commercialboundary and the set of candidate times includes at least a firstcandidate time; determine a probability based at least in part oninformation associated with the first candidate time in the context ofat least one member of a set of identified commercial boundaries;determine whether the probability satisfies a threshold; and include thefirst candidate time in the set of identified commercial boundaries whenthe probability is determined to satisfy the threshold.
 21. A computerprogram product for processing an audiovisual signal containingaudiovisual content, the computer program product being embodied in atangible computer readable storage medium and comprisingcomputer-executable instructions for: identifying a set of candidatetimes in the audiovisual content contained in the audiovisual signal,wherein each of the set of candidate times corresponds to a potentialcommercial boundary and the set of candidate times includes at least afirst candidate time; determining a probability based at least in parton information associated with the first candidate time in the contextof at least one member of a set of identified commercial boundaries;determining whether the probability satisfies a threshold; and includingthe first candidate time in the set of identified commercial boundarieswhen the probability is determined to satisfy the threshold.